I guess you could have a dedicated job dispatching task that does

while True:
    await sleep(...)
    job = await q.get()
    asyncio.ensure_future(job)

I guess the tricky bit then would be figuring out when to shut everything down... the problem is you don't know ahead of time how many jobs you'll need...

I was thinking that something like trio's nurseries would be good for that since they do the babysitting for all of the tasks (so adding the task could happen through the context manager that would be passed around)… I didn't use trio because it doesn't integrate with aiohttp and I thought sockets were way overpowered for this task.

This really seems like there should be a ready, nice pattern from somewhere in the concurrent programming universe…isn't part of a server's job being able to handle the case where you don't know ahead of time how many jobs you'll need?

@mpacer, happy to explain the different checking steps of download_check_and_move. for the most part, these are done on the new article files right after they've been downloaded to the temporary directory. also note that a number of these functions will change slightly in my current feature branch that brings the Article class into plos_corpus functions.

1. check_for_corrected_articles: for the newly downloaded articles, are any of them the 'correction' type? (In the new version, I'm changing this to three "amendment" article types that refer to other PLOS articles: correction, retraction, and expressions of concern). the idea is that for this article type, it implies that the PLOS article in the related-article field with the appropriate related-article-type may be updated in some way. It's used to create a list of articles to check if their XML content has been updated since it was last downloaded. Those articles are likely to be in corpusdir and not the temp download folder, depending on how long since the last update was run.

2. download_corrected_articles: for the list of articles derived in step 1, compare the XML on the server with the local copy. if there's a difference, download the server version into the temp folder.

3. download_vor_updates: for 5 of the 7 PLOS journals, an 'early version' or uncorrected-proof of the article is published before the final version replaces it in-place at the same DOI (version-of-record, or 'VOR'). allofplos maintains a text list of DOIs generated from local articles of uncorrected proofs (uncorrected_proofs_text_list). This function reads in that text list and checks the proof status (uncorrected vs VOR) in the remote XML file. If it's now a VOR, it downloads the new one to the tempdir and removes that DOI from the uncorrected proof list in memory.

4. check_for_uncorrected_proofs: this checks every file in the tempdir to see if it's an uncorrected proof. if it is, its DOI is added to the uncorrected proof list in memory. That list is then written back to uncorrected_proofs_text_list.

So 1 and 4 are standard checks that could be pretty easily incorporated into an async workflow, whereas 2 and 3 are checking a different set of files and/or downloading or comparing files than are likely not to be in the temporary directory.

@mpacer Could you resolve the merge conflicts?

I'm not too concerned with the merge conflicts as I don't want this to be merged yet. It's still a long way off & I am going to need a lot more time to think about how to approach this. When I push more commits I'll definitely fix the conflicts, but I'll wait until then so that I fix all the merge conflicts all at once. That's generally how I approach slow moving WIP PRs and merge conflicts since it only matters at the point when it's no longer WIP.